Hopfield networks are associative memory (AM) systems, designed for storing and retrieving patterns as local minima of an energy landscape. In the classical Hopfield model, an interesting phenomenon occurs when the amount of training data reaches its critical memory load $- spurious\,\,states$, or unintended stable points, emerge at the end of the retrieval dynamics, leading to incorrect recall. In this work, we examine diffusion models, commonly used in generative modeling, from the perspective of AMs. The training phase of diffusion model is conceptualized as memory encoding (training data is stored in the memory). The generation phase is viewed as an attempt of memory retrieval. In the small data regime the diffusion model exhibits a strong memorization phase, where the network creates distinct basins of attraction around each sample in the training set, akin to the Hopfield model below the critical memory load. In the large data regime, a different phase appears where an increase in the size of the training set fosters the creation of new attractor states that correspond to manifolds of the generated samples. Spurious states appear at the boundary of this transition and correspond to emergent attractor states, which are absent in the training set, but, at the same time, have distinct basins of attraction around them. Our findings provide: a novel perspective on the memorization-generalization phenomenon in diffusion models via the lens of AMs, theoretical prediction of existence of spurious states, empirical validation of this prediction in commonly-used diffusion models.

Since the milestone study by Alex Krizhevsky and colleagues in 2012 [1], Deep Learning (DL), with particular emphasis on Convolutional Neural Networks (CNNs), is the state-of-the-art method for image classification in many different applications. Besides classification performance, the reason for the success of CNNs is twofold: i) the recent boost of graphical processing units (GPUs) and parallel processing, that allows to train very large models; ii) the ever-growing availability of massive annotated task-specific datasets. Nonetheless, in many realistic applications many concerns may be raised about the reliability of such datasets both in terms of image and labelling quality, and consequently on the robustness of the CNN models trained and tested on them. As regards to image quality, standard CNNs are supposed to be fed with high quality samples. Nevertheless, in practical scenarios different kinds of image degradation can heavily affect the performance of a CNN both in the training and in the inference phase. Problems concerning image acquisition devices, poor image sensor, lighting conditions, focus, stabilization, exposure time or partial occlusion of the cameras may lead to produce low quality samples, which have been demonstrated to be one of the chief reasons for troublesome learning procedures of CNN models in many applications [2, 3, 4]. On the other hand, even though the CNN had been proficiently trained and validated on high quality data, noisy inputs can heavily affect the inference phase, and cause classification errors at run-time. A typical example are self-driving cars, where a partial occlusion of the image acquisition device may lead to misinterpret a road sign, with catastrophic consequences. In such settings, the well-known limitations of standard CNNs to broadcast information about how much the given input resembles the ones the model was trained on - and hence, whether the associated prediction should (or should not) be trusted - is also playing a major role.

Traditional wisdom in generative modeling literature is that spurious samples that a model can generate are errors and they should be avoided. Recent research, however, has shown interest in studying or even exploiting such samples instead of eliminating them. In this paper, we ask the question whether such samples can be eliminated all together without sacrificing coverage of the generating distribution. For the class of models we consider, we experimentally demonstrate that this is not possible without losing the ability to model some of the test samples. While our results need to be confirmed on a broader set of model families, these initial findings provide partial evidence that spurious samples share structural properties with the learned dataset, which, in turn, suggests they are not simply errors but a feature of deep generative nets.

In many applications, training data is provided in the form of related datasets obtained from several sources, which typically affects the sample distribution. The learned classification models, which are expected to perform well on similar data coming from new sources, often suffer due to bias introduced by what we call `spurious' samples -- those due to source characteristics and not representative of any other part of the data. As standard outlier detection and robust classification usually fall short of determining groups of spurious samples, we propose a procedure which identifies the common structure across datasets by minimizing a multi-dataset divergence metric, increasing accuracy for new datasets.